Water from Two Rocks: Maximizing the Mutual Information

Our goal is to forecast ground truth Y using two sources of informationXA,XB , without access to any data labeled with ground truth. That is, we are aiming to learn two predictors/hypotheses P ∗ A, P ∗ B such that P ∗ A(XA) and P ∗ B(XB) provide high quality forecasts for ground truth Y , without labeled data. We also want to elicit a high quality forecast for Y from the crowds and pay the crowds immediately, without access to Y . We build a natural connection between the learning question and the mechanism design question and deal with them using the same information theoretic approach. Learning With a natural assumption—conditioning on Y , XA and XB are independent, we reduce the learning question to an optimization problem maxPA,PB MIG (PA, PB) such that solving the learning question is equivalent to picking the P ∗ A, P ∗ B that maximize MIG(PA, PB)—the f -mutual information gain between PA and PB . We also design another family of optimization goals—PS-gain—based on the family of proper scoring rules. We can also reduce the learning problem to the PS-gain optimization problem. We will show a special case of the PS-gain—picking PS as the logarithmic scoring rule LSR—corresponds to the maximum likelihood estimator method. The range of applications of PS-gain is more limited when compared with the range of applications of f -mutual information gain. Moreover, we apply our results to the “learning with noisy labels” problem to learn a predictor that forecasts the ground truth label rather than the noisy label with some side information, without pre-estimating the relationship between the ground truth labels and noisy labels. Mechanism design We design mechanisms that elicit high quality forecasts without verification and have instant rewards for agents by assuming the agents’ information is independent conditioning on Y . In the single-task setting, we propose a forecast elicitation mechanism where truth-telling is a strict equilibrium; in the multi-task setting, we propose a family of forecast elicitation mechanisms where truth-telling is a strict equilibrium and pays better than any other equilibrium.